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ABSTRACT 

A r»atrix model for the collection, management, and 
utilization of data was developed at Eastern Iowa Community College 
District, (Davenport, Iowa) for evaluating institutional 
effectiveness. The model was examined in relation to various 
assessment instruments and the evaluation of functional areas and 
programs. Surveys of six different target groups were conducted for 
each program under review; these groups were: program administrators; 
part-time faculty; currently enrolled students; graduates of the 
program during the last three yoars; program advisory council 
members; and employers of program graduates. The process of 
constructing the matrix required a thorough examination of each 
survey instrument, the identification of items common to the various 
surveys, and the determination of uniformity in measurement of those 
various functional areas or components. This resulted in the design 
of two matrices: the Program Evaluation matrix and the Student Survey 
matrix. The matrix model facilitated integration of data from a 
variety of survey instruments into a composite report on any given 
service or functional area. An additional outcome of the matrix model 
was that it provided a rationale fo*" the revision of survey 
instruments. Contains four references. (GLR) 
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The Eastern Iowa Community College District 
Data Collection Matrix Model: 
A Tool for Functional Area and Program Evaluation 

Introduction 

Increasing emphasis on the assessment of inst iv ut ional 
effectiveness in recent years has stimulated a flurry of 
discussion and action at the institutional, state, accrediting 
agency, and federal levels. Both internal and external forces 
have contributed to the interest in formulating a comprehensive 
model to assess institutional effectiveness; such a model might 
include a variety of means of assessment and evaluation of 
institutional processes and outcomes. 

In many institutions, the evaluation of programs and 
administrative/support units, as well as the completion of other 
internal and state/federal reports have been routinely conducted 
for many years; all of these activities might be appropriate 
components of a comprehensive institutional effectiveness model. 
Therefore, it makes .sense for institutions to take an inventory of 
their existing processes and reports before devising an entirely 
new set of mea<?ures, instruments, and procedures solely for the 
assessment of institutional effectiveness. Indeed, Nichols (1989) 
advises institutions undertaking the establishme . of an 
institutional effectiveness process to examine existing data 
sources, surveys, and reports as a first step to discover the 
measures of institutional effectiveness that already exist before 
proceeding to develop new measures. 
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Apparently that str tegy has been adopted by many 
institutions in their development of institutional effectiveness 
processes. In a survey of 33 large research universities, Ory & 
Parker (1989) found that many are responding to mandates for 
institutional effectiveness information bj "repackaging" or 
"renaming" their existing programs as assessment activities. 
Rogers & Gentlemann (1989) also found that many institutions are 
already engaged in a variety of planning and evaluation processes 
that comprise a comprehensive assessment of institutional 
effectiveness . 

Hence, in attempting to formulate a comprehensive plan to 
assess institutional effectiveness, the Eastern Iowa Community 
College District (EICCD) started by conducting an inventory of its 
existing evaluation, assessment, and student follow-up processes. 
Like many other institutions, the Eastern Iowa Community College 
District conducts a variety of surveys as a part of its program 
evaluation process, its assessment of students currently enrolled 
at its colleges, and its follow-up of former students. Each of 
these surN/'eys collects information regarding various program 
components and services, as well a:5 functional areas of the 
colleges . 

Each report summarizing the survey results is issued 
independently of other survey reports. The information about any 
specific service, component, or functional area is scattered 
throughout the reports of various assessment and survey 
instruments. This phenomenon has made it extremely difficult for 
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adiTiinistrators and decision-makers to have a clear picture of any 
given functional area. 

The institutional research staff undertook to develop an 
evaluation model applicable to services and functional areas 
within the organization as one component of its institutional 
effectiveness plan. As an alternative to developing an entirely 
new set of evaluation instruments, the staff developed a matrix to 
integrate the data regarding specific functional areas from the 
numerous existing assessment sources. 

Method 

Develoomfint of the matrirps 

As a part of its Program Evaluation Process (Friedel & Papik, 
1986) , the EICCD conducts surveys of six different target groups 
for each program under review. These survey groups are as 
follows: program administrators, part-tire faculty^ currently 
enrolled students^ graduates of the program during the last three 
years^ program advisory council members^ and employers of program 
graduates. Full-time faculty conduct a self-study as a component 
of the program evaluation process, and therefore are not currently 
included in the target groups to be surveyed with a paper and 
pencil instrument . 

In addition; surveys of current students, recent graduates^ 
non-returning students, and former students are conducted on a 
egular basis. Each of these surveys asks the respondents to rate 
or evaluate severctl different functional areas and services such 
as the library^ job placement services^ advising^ counseling, and 
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registration, as well as program components such as instructor 
availability, inst ractional resources, equipment, and 
instructional materials. 

ThG items fron these various surveys comprised the' data for 
the construction of the data collection matrix. The process of 
constructing the matrix required a thorough examination of each 
survey instrument, the identification of items common to the 
various surveys, and the determination of uniformity in 
measurement of those various functional areas or components. This 
examination resulted in the design of two matrices that depict the 
items across a variety of surveys which evaluate a given function. 
Applications of the matrix 

The primary result of the development of the matrices was to 
facilitate providing a cross-survey presentation of the 
evaluations of various functional service areas to the directors 
and staff of those services areas. In addition, the matrices were 
examined in detail to discover gaps in the content areas of the 
various surveys, discrepancies in the measurement scales for 
similar items, and items that might benefit from modification 
resulting in greater consistency across instruments. The matrix 
model was tested by applying the student survey matrix to the 
functional area of job placement services. 

Results 

The matrirp.; 

Because the focus of the two sets of surveys differ, two 
matrices wera initially developed, a Program Evaluation matrix and 
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a Student Survey matrix. It is anticipated that further 
refinement of the matrix model may result in a combined matrix 
that will encompass both sets of surveys. Indeed, the two 
matrices share two major categories of items, while differing on 
the remaining two categories. 

The Program Evaluation matrix is shown in Table 1. As can be 
seer, the items from the six program evaluation surveys fall into 
four major groups: background, process items, outcome items, and 
general ratings. The process items are further subdivided into 
curriculum, faculty, resources, student & support services, and 
general process items. The outcome items are further subdivided 
into general outcomes, job placement/transfer services, and 
skills . 

The Student Surveys matrix is shown in Table 2. As can be 
seen, the items from the four student surveys also fall into four 
major groups: background, instructional items, student services 
items, and general ratings. The instructional items are further 
subdivided into items relating to curriculum, students* 
educational plans, current educational status, and occupational 
status. The student services items are further subdivided into 
activities, admissions/registration, counseling, financial aid, 
job placement, and general student services. 
Test application of the model 

The first test of the model was conducted on the functional 
area of job placement. The first level of examination is the 
survey instruments themselves. Survey respondents are asked for 
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Have opportunity to develop 
Available to help, 

good relationship with students 
Institutional relationships, leadership 
Advising 
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r.InuJnil °^ resouroes including faoUities, instructional materials, equipnient, library 

resources, and textbooks were rated on the following dimensions ^ Aiorary 

Adequacy 57 

Availability . \l ^7 
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Maintenance - - 90 

Not gender biased - - - 27 
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Adequate time for coordination 
Etfeotive interaction with other stsff 
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Table 2 

STUDENT SURVEVS - ITEM MATRIX 

This matrix shovs th« itto numbtr fron tach of the several survey instruToents which are sent 
to current and foroer students. 7he surveys are ae follows. 
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Student Data Questionnaire, adoinietered in the- third week o£ claesee 

Non-returning Student Survey, sent to students who did nut return from the 
previous seoester 

Graduate Survey, sent each fall to itudente who graduated in the previous 
fiecal year 

Follow-up Survey, ttnt %ach fall to etudents who have not been enrolled for a 
full year 
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Wage information 
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an overall rating of job placement services on both the current 
student survey (Student Data Questionnaire [SDQ]) and on the 
Follow-up Survey. Both of these ratings are on a five-point 
scale^ making for comparability on that item. Respondents to the 
Graduate Survey are asked to indicate whether or not they had 
received information on employment opportunities from anyone at 
the college. These respondents are also asked to indicate either 
rhe source of any job information or the reason for not receiving 
information. Job placement services are not rated on the Non- 
returning Student Survey at this time. 

Once the questions being asked of the various groups are 
clear, the next step in the application of the matrix is to 
compare the ratings of that functional area by the different 
respondent groups. Overall ratings of job placement services (on 
a 5-point scale, with 5 as high, 1 as low, and 3 as neutral) were 
as follows: 

Fall 1988 ^'^Q 3.21 Fall 1989 SPQ 3.26 

Follow-up 1987 3.11 Follow-up 1988 3.16 

(The SDQ is administered during the third week of the Fall 
semester each year. The Follow-up 1987 was sent in the Fall *: f 
1988 to students whose last term of enrollment was during Fiscal 
Year 1987; the Follow-up 1988 was sent in the Fall of 1989 to 
students whose last term of enrollment was during Fiscal Year 
1988.) 
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The percentage of graduates who indicated on the Graduate 
Survey that they had received inf ormat ioi. on employment 
opportunities were as follows: 

Graduate 1988 33% Graduate 1989 39% 

It is clear from this comparison of the ratings of job 
placement services across the three surveys which include this 
functional area that the ExCCD needs to seriously consider 
revising the Graduate Survey, This revision must include an 
overall rating of job placement services in order to make these 
responses comparable to the responses received on the other two 
surveys. Consideration may also be given to the possibility of 
asking directly about the availability of job placement 
information, sources, and reasons for not receiving information of 
other respondent groups. 

Comparison of the data obtained on the two similar items 
shows that students who have been away from the institution for at 
least a year rate the job placement lower than do those who are 
currently enrolled, i:e may also observe that the job placement 
services ratings of both current students and former students have 
increased in the past year. 
Additional outnnmf^fi 

As stated above, the initial impetus to the development of 
the data collection matrix was the more efficient and effective 
dissemination of data collected about specific functional areas to 
the administrators and staff of tho5e areas. A secondary benefit 
of the process has been the revision and refinement of the actual 
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surveys. This refinement was made possible by the discovery of 
differences among instruments in evaluative scales or item 
wording, as well as the inadvertent omission of specific services 
from certain survey instruments. The process has resulted in a 
more comprehensive, coordinated system of functional area 
evaluation and program review. 

The Eastern Iowa Conjnunity College District has recently 
undertaken a major project to review its transfer program 
offerings. The Program Evaluation matrix has proved to be a very 
useful summary of the existing program evaluation survey 
instruments which have previously been applied to the occupational 
programs within the District. The Program Evaluation matrix which 
summarizes the existing survey instruments has served as a 
springboard for the development of a series of survey instruments 
designed to evaluate the transfer programs. 

Discussion 

These matrices provide a snapshot view of how a given 
functional area is evaluated by a variety of groups. The matrix 
facilitates integration of data from a variety of survey 
instruments into a composite report on any given service or 
functional area. Thus, the data collection matrix allows for the 
comprehensive presentation of data collected from diverse groups 
on separate surveys relevant to any specific service or component, 
resulting in a coordinattid and unified picture useful for 
evaluation of the functional area. 
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An additional outcone of the mtrix model is that it has 
provided a rationale for the revision of survey instruments. It 
has provided an opportunity for the re-examination of the 
reasoning for asking certain questions of each group of 
respondents within each set of survey instruments. 

A comprehensive view of the various functional are , for the 
determination of institutional effectiveness can be facilitated by 
requesting similar information and perceptions from more than one 
group of respondents. This multidimensional strategy allows for 
comparison among the perceptions of various groups of constituents 
and for the validation of the perceptions of one group against 
those of another. For instance, the rating.-; of their own level of 
skill in various areas by graduates of a program can be compared 
to the employers' ratings of the level of the same skills 
Another example of appropriate comparison of responses is the 
perception of the program's image by program administrators 
compared to the image perceived by graduates, by advisory 
committee members, and by employers of program graduates. 

This effort to combine items from a variety of sources has 
resulted in a matrix template which can be user^ by other 
institutions as a guide to the construction of their own unique 
data collection matrix. Other institutional researchers will be 
able to utilize the model to examine their own survey instruments 
to develop a matrix urique to their organizational structure and 
individual needs. 
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